Method and apparatus with selective combined authentication

ABSTRACT

A method and apparatus with selective combined authentication performs a single authentication based on a first modality among plural modalities, and in response to the single authentication having failed, determines whether to perform a combined authentication by a combination of two or more of the plural modalities, and selectively, depending on a result of the determining of whether to perform the combined authentication, performs the combined authentication.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC § 119(a) of KoreanPatent Application No. 10-2018-0018666 filed on Feb. 14, 2018, KoreanPatent Application No. 10-2018-0028707 filed on Mar. 12, 2018, andKorean Patent Application No. 10-2018-0094439 filed on Aug. 13, 2018 inthe Korean Intellectual Property Office, the entire disclosures of allof which are incorporated herein by reference for all purposes.

BACKGROUND 1. Field

The following description relates to a method and apparatus for withselective combined authentication.

2. Description of Related Art

With the development of various mobile devices including smart phones,and user devices such as wearable devices, secure authentication isbeing implemented. Biometric recognition may provide a security levelfor a user device and enable a relative level of safe use of variousapplication programs such as a mobile payment application.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In one general aspect, a processor-implemented authentication methodincludes performing a single authentication based on a first modalityamong plural modalities, and in response to the single authenticationhaving failed, determining whether to perform a combined authenticationby a combination of two or more of the plural modalities, andselectively, depending on a result of the determining of whether toperform the combined authentication, performing the combinedauthentication.

The determining of whether to perform the combined authentication mayinclude determining whether a second condition for the combinedauthentication is satisfied, where the second condition may be differentfrom a first condition determinative of success or failure of the singleauthentication.

The second condition may be determined differently for each differentcombination of the plural modalities.

The second condition may be determined based on a false acceptance rate(FAR) of a modality representing a highest security among the pluralmodalities.

The second condition may be determined based on a FAR of a modalityrepresenting a highest convenience among the plural modalities.

The performing of the single authentication may include determiningwhether a first feature of the first modality satisfies a firstcondition for the single authentication.

The determining of whether to perform the combined authentication mayinclude determining whether a first feature of the first modalitysatisfies a second condition different from a first condition for thesingle authentication.

The performing of the combined authentication may include determiningwhether a result of the combined authentication satisfies a thirdcondition different from the first and second conditions.

The two or more of the plural modalities may include the first modalityand a second modality that is different from the first modality, and thedetermining of whether to perform the combined authentication mayinclude determining whether a first feature of the first modality, asecond feature of the second modality, or a combination of the firstfeature and the second feature satisfies a second condition differentfrom a first condition for the single authentication.

The two or more of the plural modalities may include the first modalityand a second modality, and where the performing of the combinedauthentication may include generating a third feature by fusing a firstfeature of the first modality and a second feature of the secondmodality, in response to a determination to perform the combinedauthentication as a result of the determining of whether to perform thecombined authentication, and performing the combined authenticationbased on the third feature.

The performing of the combined authentication may include determiningwhether the third feature satisfies a third condition for the combinedauthentication.

The combination of the two or more of the plural modalities may bedetermined based on a determined security or a convenience level for thecombined authentication.

The plural modalities may include any one or any combination of a faceimage modality, a fingerprint image modality, an iris image modality, avein image modality, a palmprint image modality, a signature modality, avoice modality, a gait modality, and a DNA structure modality of a user.The two or more of the plural modalities may also be modalities selectedby a user through a user interface for the authentication method. Themethod may further include selectively updating registration informationof the user based on the modalities selected by the user.

In one general aspect, a processor-implemented authentication methodincludes determining whether to perform a combined authentication,considering a first modality and a second modality, based on any one orany combination of a first feature of the first modality and a secondfeature of the second modality, and selectively, based on a result ofthe determining, performing the combined authentication based on thefirst feature and the second feature.

The first and second modalities may be different modalities of pluralmodalities that may include a face image modality, a fingerprint imagemodality, an iris image modality, a vein image modality, a palmprintimage modality, a signature modality, a voice modality, a gait modality,and/or a DNA structure modality of a user.

The performing of the combined authentication may include generating athird feature by fusing the first feature and the second feature, andperforming the combined authentication based on the third feature.

The performing of the combined authentication based on the third featuremay include determining whether the third feature satisfies a thirdcondition for the combined authentication.

The determining may include determining whether the first feature, thesecond feature, or a combination of the first feature and the secondfeature satisfies a second condition different from a first conditionfor a performed single authentication that is based on the firstmodality or the second modality, and determining to perform the combinedauthentication in response to the second condition being satisfied.

The second condition may be determined based on a first false acceptancerate (FAR) of the first modality, a second FAR of the second modality,or a combination of the first FAR and the second FAR.

In one general aspect, a processor implemented authentication methodincludes determining whether a first entry condition corresponding to afirst combination of plural modalities is satisfied, selectively,depending on a result of the determining of whether the first entrycondition is satisfied, performing an authentication by the firstcombination, and in response to the authentication by the firstcombination having failed, determining whether a second entry conditioncorresponding to a second combination of the plural modalities issatisfied, and selectively, depending on a result of the determining ofwhether the second entry condition is satisfied, performing anauthentication by the second combination.

The first entry condition and the second entry condition may berespectively determined differently for each of the first combination ofthe plural modalities and the second combination of the pluralmodalities.

The determining of whether the first entry condition is satisfied mayinclude determining whether the first entry condition is satisfied basedon whether the first combination of the plural modalities satisfies asecond condition that may be different from a first condition for aperformed single authentication of a modality of the plural modalities.

The first combination of the plural modalities may include a firstmodality and a second modality that may be different from the firstmodality, and the determining of whether the first entry condition issatisfied may include determining whether any one or any combination ofa first feature of the first modality and a second feature of the secondmodality satisfies the second condition, and determining that the firstentry condition is satisfied in response to the second condition beingsatisfied.

In one general aspect, a processor implemented authentication methodincludes performing a single authentication based on a first modalityamong plural modalities, and selectively, dependent on the singleauthentication having failed, performing a combined authentication by acombination of the plural modalities.

The method may further include in response to the single authenticationhaving failed, performing another single authentication based on asecond modality among the plural modalities, and wherein the selectiveperforming of the combined authentication may include determiningwhether to perform the combined authentication in response to the othersingle authentication having failed.

The performing of the combined authentication may include generating athird feature by fusing a first feature of the first modality and asecond feature of a second modality among the plural modalities, andperforming the combined authentication based on the third feature.

In one general aspect, a biometric authentication method ofauthenticating a user using a first biometric modality and a secondbiometric modality which are different from each other, includesdetermining whether biometric information of the user satisfies one of afirst condition corresponding to a feature of the first biometricmodality and a second condition corresponding to a feature of the secondbiometric modality, determining whether the biometric information of theuser satisfies a combined condition, for a combined authentication,corresponding to the feature of the first biometric modality and thefeature of the second biometric modality, and determining that thecombined authentication is successful in response to a determinationthat the biometric information of the user satisfies one of the firstcondition and the second condition, and satisfies the combinedcondition.

The biometric authentication method may further include indicating thatthe user is authenticated in response to the biometric informationsatisfying the first condition or the second condition, and in responseto the combined authentication being determined successful.

The biometric authentication method may further include determiningwhether the biometric information of the user satisfies another combinedcondition, for another combined authentication, corresponding to atleast the feature of the first biometric modality and the feature of thesecond biometric modality, and determining that the other combinedauthentication is successful in response to a determination that thebiometric information of the user satisfies the other combinedcondition.

The biometric authentication method may further include indicating thatthe user is authenticated in response to: the biometric informationsatisfying the first condition or the second condition, in response tothe combined authentication being determined successful, and in responseto the other combined authentication being determined successful.

The combined condition may be a condition corresponding to a featureresulting from a fusing of the feature of the first biometric modalityand the feature of the second biometric modality.

The combined condition may be a condition combining a score calculatedbased on the feature of the first biometric modality and a scorecalculated based on the feature of the second biometric modality.

The determining of whether the biometric information of the usersatisfies the combined condition may include generating a third featureby fusing the feature of the first biometric modality and the feature ofthe second biometric modality, where the combined condition correspondsto the third feature, and determining whether the combined conditioncorresponding to the third feature is satisfied.

The determining of whether the biometric information of the usersatisfies the combined condition may include generating a third score bycombining a first score calculated based on the feature of the firstbiometric modality and a second score calculated based on the feature ofthe second biometric modality, and determining whether the combinedcondition is satisfied based on the third score.

The first biometric modality may be an iris modality, and the secondbiometric modality may be a face modality.

In one general aspect, a mobile apparatus includes a processor, and amemory storing instructions, which when executed by the processorconfigure the processor to perform a single authentication based on afirst modality among plural modalities, and in response to the singleauthentication having failed, determine whether to perform a combinedauthentication by a combination of two or more of the plural modalities,and selectively, depending on a result of the determining of whether toperform the combined authentication, perform the combinedauthentication.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a method with selective combinedauthentication.

FIG. 2 is a flowchart illustrating an example of a method with selectivecombined authentication.

FIGS. 3A and 3B illustrate an example of a first condition and a secondcondition.

FIG. 4 is a block diagram illustrating an apparatus with selectivecombined authentication.

FIG. 5 illustrates an example of an operation of a classifier thatperforms one or more selective combined authentications.

FIGS. 6A, 6B, and 7 through 11 are flowcharts illustrating examples ofmethods with selective combined authentication.

FIG. 12 illustrates an example of a user interface.

FIGS. 13A and 13B illustrate an example of a biometrics registrationprocess.

FIG. 14 illustrates an example with selective combined authentication byutilizing a correlation between different modalities.

FIG. 15 is a block diagram illustrating an example of an apparatus witha combined authentication.

Throughout the drawings and the detailed description, unless otherwisedescribed or provided, the same drawing reference numerals will beunderstood to refer to the same elements, features, and structures. Thedrawings may not be to scale, and the relative size, proportions, anddepiction of elements in the drawings may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent after an understanding of thedisclosure of this application. For example, the sequences of operationsdescribed herein are merely examples, and are not limited to those setforth herein, but may be changed as will be apparent after anunderstanding of the disclosure of this application, with the exceptionof operations necessarily occurring in a certain order. Also,descriptions of features that are known in the art may be omitted forincreased clarity and conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

Although terms such as first, second, A, B, (a), (b), and the like maybe used herein to describe components, the components are not limited tothe terms. In addition, each of these terminologies is not used todefine an essence, order, or sequence of a corresponding component butused merely to distinguish the corresponding component from othercomponent(s). For example, a “first” component may be referred to as a“second” component, or similarly, and the “second” component may bereferred to as the “first” component within the scope of the rightaccording to the concept of the present disclosure.

It should be noted that if it is described in the specification that onecomponent is “connected,” “coupled,” or “joined” to another component, athird component may be “connected,” “coupled,” and “joined” between thefirst and second components, although the first component may bedirectly connected, coupled or joined to the second component. Inaddition, it should be noted that if it is described in thespecification that one component is “directly connected” or “directlyjoined” to another component, a third component may not be presenttherebetween. Likewise, expressions, for example, “between” and“immediately between” and “adjacent to” and “immediately adjacent to”may also be construed as described in the foregoing.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used herein, thesingular forms “a,” “an,” and “the,” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willbe further understood that the terms “comprises,” “comprising,”“includes,” and/or “including,” when used herein, specify the presenceof stated features, integers, operations, elements, and/or components inan example embodiment, but do not preclude the presence or addition ofone or more other features, integers, operations, elements, components,and/or groups thereof in alternative embodiments, nor the lack of suchstated features, integers, operations, elements, components, and/orcombinations/groups thereof in further alternative embodiments unlessthe context and understanding of the present disclosure indicatesotherwise. The use of the term ‘may’ herein with respect to an exampleor embodiment, e.g., as to what an example or embodiment may include orimplement, means that at least one example or embodiment exists wheresuch a feature is included or implemented while all examples andembodiments are not limited thereto.

Unless otherwise defined, all terms, including technical and scientificterms, used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure pertains based onan understanding of the present disclosure. Terms, such as those definedin commonly used dictionaries, are to be interpreted as having a meaningthat is consistent with their meaning in the context of the relevant artand the present disclosure, and are not to be interpreted in anidealized or overly formal sense unless expressly so defined herein.

Examples set forth hereinafter include any of various types of productssuch as a personal computer, a laptop computer, a tablet computer, asmart phone, a television, a smart home appliance, an intelligentvehicle, a kiosk, and a wearable device. For example, the examples maybe, or applied to, a smart phone, a mobile device, a smart home system,an intelligent vehicle, and an automated teller machine (ATM) for userauthentication, and the apparatuses and methods demonstrated in theaccompanying drawings are thereby representative of the same.Accordingly, the examples will be described in detail with reference tothe accompanying drawings, wherein like drawing reference numerals areused for like elements.

FIG. 1 illustrates an example of a method with selective combinedauthentication. Referring to FIG. 1, a situation in which a user 50performs a biometric recognition or authentication using an image sensor110, an infrared (IR) sensor 120, or a fingerprint sensor 130 of amobile device 100 is illustrated.

In an example, the biometric recognition or authentication may beperformed selectively using a plurality of modalities based on variousbiometrics such as a face, an iris, a fingerprint, and a vein, asnon-limiting examples. Thus, the term “modalities” refers to uniquebiometric information of a user such as, for example, a face, afingerprint, an iris, a vein, a palmprint, a signature, a voice, a gait,and a DNA structure of the user, or various aspects that representunique information that may then be used to verify, recognize, orauthenticate, among other example functions, the user, noting that suchmodality examples are not intended to be limiting and alternativeexamples are also available.

When using various biometrics, the corresponding provided security levelof the respective biometric authentication varies such verification,recognition, or authentication for each modality. For example, anauthentication using an iris modality may have a false acceptance rate(FAR) of one ten-millionth, while respective authentications using afingerprint modality and a face modality each may have a FAR of onemillionth. The term “FAR”, or false acceptance rate, refers to a rate offalsely verifying or misverifying biometric information of anotherperson as biometric information of one person, a rate of falselyauthenticating or misauthenticating biometric information of the personas biometric information of the one person, or a rate of falselyrecognizing or misrecognizing biometric information of the other personas biometric information of the one person. Herein, such FARs of suchverifying, authenticating, or recognizing using the respectivemodalities will be referred to as the respective FARs of the modalities,just as securities or security levels of such modalities are hereinreferences to securities or security levels provided or made availableby or by consideration of the corresponding modality. As a FAR of amodality with respect to corresponding biometric information decreases,a security, that is, a security level, of the modality increases.

In an example of using a combined authentication of performing averification, recognition, or authentication through a combination ofvarious modalities, a security may be improved of the respectiveseparate modality provided securities by considering respectiveadvantages and/or disadvantages of the modalities. For example, acombined authentication through the combination of the modalities maymore confidently indicate accurate authentication performance suitablefor various situations. Herein, while the terms verification,recognition, or authentication may have different meanings, e.g., withverification being a based on comparisons to a known class of personsand recognition or authentication being based on the same orconsideration of greater or multiple classes or withoutpre-classification of persons, descriptions of examples with respect toany of such verification, recognition, or authentication should beunderstood to be applicable to all of such verification, recognition,and authentication examples.

In an example, a user may be selectively authenticated using an irisimage and a face image. The iris image and the face image may becaptured through an IR camera, a color camera, a monochrome camera, or athree-dimensional (3D) camera, as non-limiting examples. The 3D cameramay include or be implemented as one or more respective various types ofcameras such as a time-of-flight (ToF) camera and a structured lightcamera, noting that such types of 3D cameras are merely demonstrativeexamples and thus 3D camera examples herein are not limited thereto. Inan example, the iris image may be captured through the IR camera oranother type of camera, e.g., a color camera, configured to also captureIR information, while the face image is captured through the colorcamera, one or more other color cameras, or the example 3D camera. Thus,the iris image and the face image may be captured through variouscombinations of the above-mentioned cameras, as non-limiting examples.

The iris modality may have a higher security than the face modality. Forexample, if a user A and a user B are twins, consideration of the irismodalities of the respective user A and user B will result in differentfeature vectors, for example, similar to those of different non-twinpersons. However, with respect to the face modality, considerations offace modalities of the respective example twin user A and user B willresult in similar feature vectors, for example, at similar to those of asame person. Thus, it may be difficult to distinguish between twinsbased on the face modality, while it may be relatively easy todistinguish the twins based on the iris modality.

However, depending on an environment of capturing, the performance ofiris modality may become lower than the performance of face modality.For example, the iris image may be captured using an IR sensor fromreflected light emitted from a light emitting diode (LED) light includedin a mobile device. However, under strong outdoor light includingnaturally occurring infrared light, the quality of the iris imagedegrades. For example, with such an iris modality as well as some or allother modalities, as the image quality degrades, the recognitionperformance decreases. Thus, in a case of using the iris modality only,the recognition performance may decrease in a specific environment, forexample, in a situation in which strong outdoor light exists.

Rather, the quality of the face image does not typically degrade in thepresence or environment of strong outdoor light. Thus, the recognitionperformance of face modality may not decrease even in a situation inwhich the recognition performance of iris modality decreases. However,unlike an example iris image, the example face image may be vulnerableto changes in type of environmental light or may vary depending on suchdifferent types of environmental light, such as a high luminancesituation type compared to a low luminance situation type, and theexample face image modality security or FAR may vary for different faceposes or changes in face pose. Here, the term environment is meant torefer to the exterior of the camera through which incident lighttravels, e.g., where the environment is an outdoor environment thecamera would be capturing incident light that is affected by light inthe outdoor scene corresponding to the location of the camera.Similarly, references to an environment of low or high or strong indooror outdoor light are references to the natural and/or artificial lightsources in the surrounding of the camera that may produce light that mayalso be incident on the camera.

Thus, in an example, the iris image and the face image may be usedcombinedly or both considered, or selectively considered, to maintain acertain or desirable recognition performance in various situations orenvironments.

The examples set forth hereinafter should be understood as alsoincluding examples that consider both the security and the userconvenience when performing a combined authentication by a combinationof modalities. For example, examples may provide technology thatimproves the user convenience while maintaining the security, ortechnology that improves the security while maintaining the userconvenience, as non-limiting examples.

In an example, security needs or desire may be considered more importantfor a payment application, e.g., compared to convenience or other needsor desires, and thus a scenario for performing a single authenticationand a combined authentication may be set to be implemented to improvethe user convenience while maintaining the security. For example,examples may primarily perform a single authentication using a modalityhaving a relatively high security, for example, an iris modality, andperform a combined authentication through a combination of a pluralityof modalities, for example, the iris modality and a face modality, inresponse to the primary authentication having failed, or a resultingscore or confidence level of the primary authentication failing to meetthe corresponding single modality success threshold. In such an example,if the primary authentication is successful, or the resulting score ofconfidence level meets the corresponding single modality successthreshold, then the combined authentication may not be performed and thecorresponding user may be authenticated. Similarly, the successful orfailed authentication of the user may be determined by one or more ofsuch combined authentications based on their respective meetings ornon-meetings of corresponding success thresholds.

In another example, user convenience needs or desire may be consideredmore important for an unlock function of a smart phone, e.g., comparedto security or other needs or desires, and thus a scenario forperforming a single authentication and a combined authentication may beset to maintain or improve the security while maintaining the userconvenience at a high level. For example, examples may primarily performa single authentication using a modality having a relatively highconvenience, for example, a face modality, and secondarily performanother single authentication using a modality having a differentfeature or biometric information from the modality used for the primaryauthentication, for example, an iris modality, in response to theprimary authentication having failed, or a resulting score or confidencelevel of the primary authentication fails to meet the correspondingprimary single modality success threshold. In response to the secondaryauthentication having also failed, or a resulting score or confidencelevel of the secondary authentication failing to meet the correspondingsecondary single modality success threshold, one or more other combinedauthentications nay be performed through various combinations of theplurality of modalities, for example, the face modality and the irismodality, may be performed. In such an example, if the primaryauthentication is successful, or the resulting score or confidence levelmeets the corresponding primary single modality success threshold, thenthe combined authentication may not be performed and the correspondinguser may be authenticated, and if the primary authentication isunsuccessful but the secondary authentication is successful, or theresulting score of confidence level meets the corresponding secondarysingle modality success threshold, then the combined authentication maynot be performed and the corresponding user may be authenticated.Similarly, with the failure of both the primary and secondaryauthentications, the successful or failed authentication of the user maybe determined by one or more of such further combined authenticationsbased on their respective meetings or non-meetings of correspondingsuccess thresholds.

Thus, one or more or all included examples herein provide technologythat together may satisfy a desired or determined, e.g., based on userselection or an example application program requirement, security anduser convenience level or importance or balance by determiningrespective ‘entry conditions’ as to whether to perform one or morecombined authentications prior to performing the corresponding combinedauthentications, i.e., the determination of whether to perform such acombined authentication can be performed prior to the performance of thecombined authentication. Accordingly, such entry conditions may bepredetermined, e.g., prior to the performance of the example primary orsecondary example authentications, based on which modalities areincluded in the combination of modalities for the combinedauthentication, the type or category of application to which thecombined authentication is being or going to be applied, and/orinformation indicated by such applications.

Thus, an entry condition to determine whether to perform the combinedauthentication may be a condition related to at least one of themodalities for the combined authentication. For example, the entrycondition may include a first condition related to the face modality, asecond condition related to the iris modality, and a combination of thefirst condition and the second condition.

In an example, the entry condition is a condition different from areference condition to determine whether a single authentication usingan individual modality is successful. For example, in a case in whichthe entry condition is related to at least the iris modality, the entrycondition may require a lowered FAR compared to a FAR of the singleauthentication using the iris modality, e.g., thereby providing highersecurity than the single authentication using the iris modality.

In an example, the combined authentication is applied to a paymentapplication. In this example, an entry condition that considers thesecurity to be more important, e.g., more important than anotherapplication or function such as unlock operation or more important thanuser convenience, while also improving user convenience may be set.Thus, in such examples where security is more important but some userconvenience still desired, the examples may determine a respective entrycondition for determining whether to perform a corresponding one or morecombined authentications based on, or primarily based on, one or moremodalities having a relatively high security in the correspondingcombined authentication, for example, the iris modality, and thenperform the corresponding combined authentication through a combinationof the plurality of modalities, for example, the iris modality and theface modality.

In another example, the combined authentication is applied to an unlockfunction of a smart phone. In this example, an entry condition thatconsiders the user convenience to be more important, e.g., moreimportant than another application of function such as the paymentapplication or more important than security, while also improvingsecurity may be set. Thus, in such examples where user convenience ismore important, the examples may determine a respective entry conditionfor determining whether to perform a corresponding one or more combinedauthentications based on, or primarily based on, a single modality, forexample, the iris modality, the face modality, or a fingerprintmodality, and then perform the combined authentication through acombination of the plurality of modalities, for example, the facemodality, the fingerprint modality, and the iris modality.

As described below, an entry condition is set for use for determiningwhether to perform a combination of single modalities, e.g., after oneor more authentication failures of respective single modalities, ratherthan being set to use for respectively determining whether to performany one of the authentications using only a single modality. Further,when performing the combined authentication, a scheme of combining theplurality of modalities may be set in various manners in differentexamples.

As noted above, respective scores or other confidence indicators foreach single authentication for a single modality may be determined, forexample, such as in the forms of matching scores or distance scores,e.g., as distances between a feature vector corresponding to the singlemodality and a registered feature vector for a registered user.Likewise, for each combined authentication considering multiplemodalities, similar respective scores or other confidence indicators maybe determined. Thus, each of such matching scores correspond to a scorerepresenting how similar a registered frame, image, series of images, orother biometric information and an input frame, image, series of images,or other biometric information are to each other, that is, a similaritybetween such considered information, i.e., the respective frames,images, series of images, and biometric information. A low matchingscore indicates a low similarity between the considered information, anda high matching score indicates a high similarity between the consideredinformation. The higher the matching score, the higher probability themobile device 100 accepts an authentication of the user 50. For example,when the matching score meets, e.g., is equal or greater than, amatching score authentication threshold, the user 50 is authenticated.Further, as noted, such distance scores may correspond to a scorerepresenting a feature distance, for example, a Euclidean distance,between the registered frame, image, series of images, or otherbiometric information and the input frame, image, series of images, orother biometric information. A low distance score indicates a shortfeature distance between the considered information, e.g., with in acorresponding feature vector space, and a high distance score indicatesa long feature distance between the considered information. Accordingly,the lower the distance score, the higher probability the mobile device100 accepts an authentication of the user 50. For example, when thedistance score meets, e.g., is equal or less than, a distinct scoreauthentication threshold, the user 50 is authenticated.

As described above, one or more or all of such various examples providetechnological improvements in computer functionality of at leastimproving a user convenience of accessing a device with more accuraterecognitions with at least selective emphases between security and userconvenience in various implementations, such as through more accuratelyguaranteeing a relatively high security through a selective combinedauthentication by a selective combination of modalities, such ascompared to previous approaches that only performed user authenticationusing a single corresponding modality. In addition, one or more or allexamples herein improve user convenience by maintaining the userrecognition and authentication performance in various environments usingvarious modalities.

FIG. 2 is a flowchart illustrating an example of a method with selectivecombined authentication. Referring to FIG. 2, in operation 210, anapparatus with selective combined authentication, hereinafter, the“authentication apparatus”, is configured to perform respective singleauthentications of individual corresponding modalities, and selectivelyperforms one or more various combined authentications of respectivemultiple corresponding modalities in response to such one or morerespective single authentications having failed. For example, theauthentication apparatus may perform the single authentication bydetermining whether a first feature of a first modality satisfies afirst condition. The first feature may be, for example, a Euclideandistance or a similarity or matching score determined based on featurevectors extracted from the first modality and stored feature vectors,for example. Further, the first condition may correspond to, forexample, a threshold distance for the single authentication or athreshold score for the single authentication. In an example, the firstcondition for the single authentication may be determined or setdifferently based on the type of the modality, such as face, iris, palm,vein, or fingerprint respective modality types, as non-limiting example.In response to a result of the single authentication being that thecorresponding condition is met in operation 210, i.e., that theauthentication is a success, the authentication apparatus determinesthat the authentication of the user is successful, in operation 240.

In response to the single authentication having failed in operation 210,i.e., the corresponding condition not have been met, the authenticationapparatus determines in operation 220 whether there is satisfaction ofan entry condition, where the entry condition is determinative ofwhether the authentication apparatus should perform a combinedauthentication by a combination of the modalities. The entry conditionis thus based on a second condition different from the first conditionconsidered for the single authentication. Here, the example “combinedauthentication” is construed as performing an authentication throughvarious combinations of modalities, for example, (fingerprint and irismodalities), (fingerprint and face modalities), (iris and facemodalities), and (fingerprint, iris, and face modalities), among aplurality of non-limiting example modalities such as the examplefingerprint, iris, and face modalities.

Thus, in operation 220, the authentication apparatus determines, withrespect to the combination of the modalities, whether to perform thecorresponding combined authentication. The authentication apparatusdetermines whether to perform the combined authentication based onwhether the combination of the modalities satisfies the secondcondition. In this example, the second condition is determineddifferently for each different combination of such modalities.

The second condition is determined, for example, based on a FAR of amodality having a highest security in the combination of the modalitiesconsidered in the combined authentication. For example, in a case inwhich the combination of the modalities includes an iris modality and aface modality, the second condition is determined based on oneten-millionth, as only an example, which may be a FAR of the irismodality, which has a higher security than the face modality forexample. The second condition may require a FAR different from the FARof one ten-millionth, i.e., as the second condition may be differentfrom a corresponding first condition of the same modality. In anexample, in a case in which the combination of the modalities includes aface modality and a fingerprint modality, the second condition may bedetermined based on one millionth, which may be a non-limiting exampleFAR of the fingerprint modality, which has a higher security than theface modality for example. The second condition may require a FAR lowerthan the FAR of one millionth, for example.

The second condition may be determined, for example, based on a FAR of amodality having a highest convenience in the combination of themodalities. For example, in a case in which the combination of themodalities includes a face modality and a signature modality, the secondcondition may be determined based on one two-thousandth, which is anon-limiting example FAR of the face modality, which has a higherconvenience than the signature modality, for example. The secondcondition may require a FAR different from the FAR of onetwo-thousandth. A method of the authentication apparatus determining thefirst condition and the second condition will be described further withreference to FIGS. 3A and 3B.

The combination of the modalities may include, for example, the firstmodality and a second modality different from the first modality. Inanother example, the combination of the modalities may include threemodalities such as the first modality, the second modality, and a thirdmodality, or the combination may include more than three modalities. Thenumber of modalities included in the combination of the modalities isdetermined, for example, based on the security or the convenience of thecombined authentication, e.g., based on whether the authentication isdetermined or set to give greater priority for security over convenienceor greater priority of convenience over security.

In operation 220, the authentication apparatus determines whether anyone or any combination of the first feature of the first modality and asecond feature of the second modality satisfies the second condition. Inresponse to the second condition being satisfied, for example, accepted,the authentication apparatus determines to perform the combinedauthentication by the combination of the modalities. In response to thesecond condition not being satisfied, for example, having failed, inoperation 220, the authentication apparatus may determine that theauthentication of the user has failed, in operation 250.

In response to a determination to perform the combined authentication inoperation 220, the authentication apparatus performs the combinedauthentication, in operation 230. In operation 230, the authenticationapparatus performs the combined authentication by determining acondition in which a score based on the first feature and a score basedon the second feature are combined. For example, the authenticationapparatus determines whether a third condition is satisfied, the thirdcondition in which a condition for the score based on the first featureand a condition for the score based on the second feature are combinedthrough a logical operation, for example.

In another example, the authentication apparatus generates a thirdfeature by fusing the first feature and the second feature, e.g.,through a logical or mathematical operation, and performs the combinedauthentication based on the third feature. For example, theauthentication apparatus performs the combined authentication based onwhether the third feature satisfies a third condition for the combinedauthentication. The third condition may be determined based on a targetFAR for the combined authentication, for example.

In operation 230, the authentication apparatus determines whether acombination of the first feature and the second feature or the thirdfeature satisfies the third condition, for example, using a pre-trainedclassifier. A method of an authentication apparatus, such as theauthentication apparatus of FIG. 2, performing the combinedauthentication based on the third feature will be described further withreference to FIG. 5.

FIGS. 3A and 3B illustrate an example of determining a first conditionand a second condition. For example, the first condition corresponds toa threshold distance for performing a single authentication, and asecond condition corresponds to a threshold distance used to determinewhether to perform a combined authentication. In this example, thethreshold distance may be determined based on a verification rate (VR),a FAR, a false rejection rate (FRR), or a combination thereof, asnon-limiting examples.

Hereinafter, a method of determining the first condition and the secondcondition using the FAR will be described for ease of description.However, the method of determining the first condition and the secondcondition is not limited thereto, and in examples the first conditionand the second condition are determined through consideration of variousperformance indices. Further, although the performance indices todetermine the first condition and the second condition are discussed inan example as being the same, target scores to determine respectivethresholds corresponding to the first condition and the second conditionmay be different from each other. For example, the thresholdcorresponding to the first condition may be determined to satisfy apredetermined FAR for a RAW score, e.g., compared to a score in whichadditional processing, for example, applied filtering, has beenadditionally applied. For example, the threshold corresponding to thesecond condition may be determined to satisfy a predetermined FAR forthe score obtained by applying such additional processing, for example,such applied filtering, to the RAW score.

Referring to FIG. 3A, a feature distance by a first feature of a firstmodality between items of data corresponding to a user, the same person,is represented using a histogram 310. Further, a histogram 330 shows afeature distance by the first feature of the first modality between datacorresponding to the user and data corresponding to another person.Here, the feature distance corresponds to a Euclidean distancerepresenting a difference level between a face image of the user and aface image of the other person. The feature distance has a relativelysmall value when a similarity between data to be compared is relativelyhigh, and has a relatively great value when the similarity between thedata to be compared is relatively low.

In an example, an authentication apparatus determines the firstcondition and the second condition based on various performance indicessuch as, and as non-limiting examples, a normalized cross correlation(NCC), a matching score, or a similarity score between feature vectors,in addition to the feature distance.

In a graph of FIG. 3A, the axis X represents a feature distance by thefeature of the first modality, and the axis Y represents the number ofsamples corresponding to the feature distance.

The authentication apparatus determines the first condition based on adetermination of whether to emphasize security or whether to emphasizeconvenience, such for accessing or performing a face verification orauthentication of a particular application to type of application. Forexample, in an example where a user executes a financial institutionapplication such as a banking or stock trading application, theauthentication apparatus determines the first condition such that astrict authentication is performed by emphasizing security, e.g.,emphasizing security over convenience. In an example where the user isattempting to perform a select or limited unlocking of a mobile device,to permit or to perform only a simple function of the mobile device suchas a camera or a memo, the authentication apparatus may determine thefirst condition such that the authentication process is omitted, or suchan authentication is performed based on a new criterion by emphasizingconvenience, e.g., emphasizing convenience over security. In anotherexample, where the user is attempting to access or perform otheroperations of the mobile device, e.g., to access personal information oraccess other operations in addition to such camera and memoapplications/functions, the authentication apparatus may determine thefirst condition such that the authentication is performed based on a newcriterion by emphasizing security, e.g., emphasizing security overconvenience.

As an example of emphasizing security, the authentication apparatus mayset a feature distance of a boundary line 350 as the first condition,where the boundary line 350 distinguishes an area of a level that wouldallow a misrecognition in an iris modality, for example, the bottom oneten-millionth, of the entire area of the histogram 330 showing a featuredistance by a first feature of an iris modality of another person. Inthis example, in response to the feature distance of the iris modalitynot satisfying the FAR of one ten-millionth, a single authenticationwith respect to the user may thus fail.

However, as noted above, the iris modality may not work well in anoutdoor environment where there are strong ultraviolet rays, and thus ifthe iris modality satisfies a predetermined condition, even if theauthentication by the iris modality fails, the iris modality may stillbe used again in the performing of the combined authentication inconjunction with the face modality, for example, which as noted aboveworks well even in an outdoor environment, which may thereby alleviateuser inconvenience. For example, in previous approaches where only theiris modality was implemented, the corresponding authentication may havea substantially higher FRR (false recognition rate) in such outdoorenvironments, and prevent access to a valid user. As an example of theabove, if the iris modality does not satisfy a feature distancethreshold of the boundary line 350 for the single authentication butsatisfies a feature distance threshold of another boundary line 370, theauthentication apparatus may determine to perform the combinedauthentication by a combination of the iris modality and the facemodality. The iris modality satisfying the feature distance thresholdindicates that a feature distance by the iris modality is less than (orequal to) the feature distance threshold.

In this example, a boundary line for the first condition and a boundaryline for the second condition are determined by different scorehistograms. For example, referring to FIG. 3B, the boundary line 350 forthe first condition is set based on the original modality score, forexample, the histogram 310 and the histogram 330, and the boundary line370 for the second condition is set based on a score obtained byapplying an additional processing, for example, filtering to thecorresponding score, for example, the histogram 320 and the histogram340.

In an example, if the iris modality satisfies the second conditionalthough the single authentication by the iris modality has failed, theauthentication apparatus may configure the iris modality as an elementof the combined authentication, that is, a combination of modalities,thereby improving the user convenience while maintaining security. Infurther detail, in response to the single authentication having failed,by setting an entry condition, for example, the second condition, ratherthan immediately performing the combined authentication, a rate ofmisrecognizing another person as one person may be reduced. That is, thecombined authentication may be performed only in response to the entrycondition, for example, the second condition, being satisfied, and thus,a relatively high security may be maintained when compared to a case ofnot using the entry condition, for example, the second condition.Further, compared to a case of using the first condition only, a rate ofincorrectly excluding one person as another person decreases in examplesherein, and thus user convenience may improve over using the firstcondition only.

FIG. 4 is a block diagram illustrating an apparatus with selectivecombined authentication, for example. Referring to FIG. 4, anauthentication apparatus includes an enrollment database (DB) 410, amatcher 420, an entry condition determiner 430, and an authenticator440. Operations of the matcher 420, the entry condition determiner 430and the authenticator 440 may be processor elements or components ofprocessor 450, or respective operations of the same performed by aprocessor 1510 of FIG. 15, for example, which will be described later.

The enrollment DB 410 includes enrollment feature vectors for eachmodality. For example, the enrollment DB 410 may be provided for each ofmodalities or provided as a single unified DB including all themodalities. The enrollment DB 410 may be, or stored in, a memory of theauthentication apparatus, and thus, such enrollment feature vectors maybe requested and/or accessed from the enrollment DB 410 in the memory.

The matcher 420 my thus request or call in real time, for example, andfrom the enrollment DB 410, an enrollment feature vector of each of afirst modality (Input1) and a second modality (Input2) being input.Here, the input 1 and the input 2 are representative of the respectivefirst and second modality information, e.g., where the input 1 is acolor face image and input 2 is an infrared iris image.

The matcher 420 extracts a feature vector of the first modality and/or afeature vector of the second modality, and calculates a feature distanceor a similarity score by matching the feature vector of the firstmodality and/or the feature vector of the second modality with theenrollment feature vectors of the corresponding modalities stored in theenrollment DB 410. In an example, the matcher 420 may extract, in viewof an authentication situation of the authenticator 440, the featurevector of the first modality and then extract the feature vector of thesecond modality as necessary, rather than extracting the feature vectorsof the first modality and the second modality at the same time, thoughin another example the feature vectors may be respectively extracted asthe same time.

The matcher 420 transmits the determined feature distance or similarityscore corresponding to a matching result of the matcher 420 to the entrycondition determiner 430 and the authenticator 440.

For example, in response to the feature distance between the featurevector of the first modality received from the matcher 420 and theenrollment feature vector satisfying a first condition, theauthenticator 440 determines that a single authentication is successfuland outputs “single authentication success”. In response to the singleauthentication being successful, the authentication apparatus outputs“authentication accept”. In response to the feature distance between thefeature vector of the first modality and the enrollment feature vectornot satisfying the first condition, the authenticator 440 determinesthat the single authentication has failed and outputs “singleauthentication failure”. The authenticator 440 transmits anauthentication result to the entry condition determiner 430.

The entry condition determiner 430 determines whether an entry conditionfor a combined authentication is satisfied based on the matching resultreceived from the matcher 420, for example, a single authenticationsuccess or a single authentication failure. In response to the singleauthentication failure being received from the matcher 420, the entrycondition determiner 430 determines whether the entry condition for thecombined authentication is satisfied.

The entry condition determiner 430 then determines whether to performthe combined authentication by a combination of the modalities, based ona second condition different from the first condition for the singleauthentication. For example, in response to information indicating thata feature distance between at least one of a first feature vector of thefirst modality and a second feature vector of the second modality andthe respective enrollment feature vector of the corresponding modalitiesstored in a DB satisfies the second condition being received from thematcher 420, the entry condition determiner 430 determines that theentry condition for the combined authentication is satisfied. Inresponse to a determination that the entry condition for the combinedauthentication is satisfied, the entry condition determiner 430 requeststhe authenticator 440 to perform the combined authentication by thecombination of the modalities. In response to a determination that theentry condition for the combined authentication is not satisfied, theauthentication apparatus outputs “authentication failure”.

The authenticator 440 performs the combined authentication based on acondition in which a score based on a first feature and a score based ona second feature are combined. In another example, the authenticator 440generates a third feature by fusing the first feature and the secondfeature, and performs the combined authentication based on the thirdfeature. The authenticator 440 outputs a result of performing thecombined authentication, for example, combined authentication success orcombined authentication failure. The authentication apparatus outputsauthentication success in response to the combined authentication beingsuccessful, and outputs authentication failure in response to thecombined authentication having failed. The authenticator 440 performsthe combined authentication, for example, by a classifier 500 which willbe described later with reference to FIG. 5. A process of theauthenticator 440 performing the combined authentication based on thethird feature will be described further with reference to FIG. 5.

FIG. 5 illustrates an example of an operation of a classifier that isconfigured to perform one or more selective combined authentications.Referring to FIG. 5, the classifier 500 may perform a combinedauthentication based on a third feature generated by fusing a firstfeature of a first modality and a second feature of a second modality,such as the first feature of the first modality and the second featureof the second modality discussed above with respect to FIG. 4.

For example, in a non-limiting example, 16-dimensional (16D) featurevectors of an iris image and 3-dimensional (3D) feature vectors of aface image may each be input into the classifier 500.

The classifier 500 may then generate 19-dimensional (19D) featurevectors by fusing the 16D feature vectors of an iris modality and the 3Dfeature vectors of a face modality. In this example, the 16D featurevectors of the iris modality may be feature vectors corresponding tofeatures such as a Hamming distance of an iris, a bit count of the iris,a radius of the iris, a shape and a color of the iris, and morphemes ofretinal capillaries, as non-limiting examples. Further, the 3D featurevectors of the face modality may be feature vectors corresponding tofeatures of an entire face or a partial face, also as non-limitingexamples.

In this example, the classifier 500 is configured to determine whetherthe combined authentication has been successful or not, i.e., whether itis accepted or failed, by comparing the 19D feature vectors to aboundary line or a reference value that distinguishes one person fromanother person. For example, the classifier 500 may be a trainedclassifier.

A graph on a lower left side of FIG. 5 shows a learning phase of theclassifier 500. In the graph showing the learning phase, the axis Xrepresents the example 16D feature space of the example iris image, andthe axis Y represents the example 3D feature space of the face image.For ease of description, the 3D feature space of the example face imagewill be described as an example, and the 16D feature space of theexample iris image will be described as an example, noting thatrespective examples are not limited thereto. For example, in an example,the feature space may be extended to a 20-dimensional (20D) featurespace in which a separate axis exists for each dimension, such as byincreasing by 1 the dimension of the face image.

In FIG. 5, the illustrated dots in the shown learning phase correspondto the 19D vectors generated by the fusing of the 16D feature vectorsand the 3D feature vectors. The boundary line 510 corresponds to aparameter that the classifier 500 ultimately learns for determiningwhether a combined authentication by the 19D vectors is accepted orfailed. The classifier 500 learns the parameter or a correspondingweight with respect to feature vectors, for example, using a supportvector machine (SVM). In this example, the parameter that the classifier500 learns is determined based on a level of a target FAR of theauthentication apparatus, though examples are not limited thereto.

A graph on a lower right side of FIG. 5 shows an authentication phase ofthe classifier 500. In the authentication phase, the boundary line 530corresponds to a parameter of the classifier 500. The line 530corresponds to a simplified two-dimensional (2D) representation of the19D feature vectors. For example, in response to the fused 19D featurevectors being generated, the authentication apparatus may perform acombined authentication with respect to the 19D inputs based on theboundary line 530. For example, the authentication apparatus maydetermine a first input (I₁, F₁) on the left side of the boundary line530, e.g., below the boundary line 530, to be authentication accept, anddetermine a second input (I₂, F₂) on the right side of the boundary line530, e.g., above the boundary line 530, to be authentication fail.

FIG. 6A is a flowchart illustrating an example of a method withselective combined authentication. Referring to FIG. 6A, a process withselective combined authentication with respect to iris datacorresponding to a first modality and face data corresponding to asecond modality is illustrated.

An authentication apparatus performs the combined authentication throughoperation 610 of extracting feature information of a modality, andoperation 650. The authentication apparatus further performs operation630 of determining whether to perform the combined authentication basedon a security level. In an example, operations 630 and 650 are performedin order, while in another example the operations 630 and 650 areperformed out of order.

In operation 610, in response to the iris data and the face data beinginput, e.g., in response to the iris data and the face data beingrespectively captured by respective cameras of the authenticationapparatus or requested or received from such respective cameras, orstored to and read from memories or buffers in which such data maytemporarily be stored, the authentication apparatus extracts a featurevector with respect to each of modalities. The authentication apparatuscalculates an iris score or a face score by performing a matching orsimilarity determination operation between the extracted feature vectorswith enrollment feature vectors for each corresponding modality storedin an enrollment DB 605. In this example, the iris score or the facescore corresponds to a feature distance or a similarity score betweenthe feature vector extracted from each of the modalities and theenrollment feature vector.

In operation 650, the authentication apparatus determines whether one ofthe iris score or the face score satisfies a single authenticationcondition. For example, in one circumstance the authentication apparatusdetermines whether the iris score satisfies a corresponding singleauthentication condition, while in another circumstance theauthentication apparatus determines whether the face score satisfies acorresponding single authentication score. In response to the singleauthentication condition being satisfied, the authentication apparatusaccepts an authentication.

In response to the either of the example iris score or the face scorenot satisfying the corresponding single authentication condition, theauthentication apparatus determines whether a face score and/or an irisscore satisfies a predetermined FAR based on a set security level, inoperation 630. In detail, whether the face score separately satisfiesthe predetermined FAR, whether the iris score separately satisfies thepredetermined FAR, or whether both the face score and the iris scoresatisfy the predetermined FAR may be used to determine whether thepredetermined FAR is satisfied.

The predetermined FAR is determined to be a FAR for authentication ofthe face data or a FAR for authentication of the iris data, based on theset security level. In response to the combined modality score, e.g.,based on the resultant face modality score and iris modality score, orrespectively both resultant face and iris modality scores, notsatisfying the predetermined FAR in operation 630, the authenticationapparatus acquires new iris data and/or face data and performs thecombined authentication process of FIG. 6A again.

In response to (face score, iris score) satisfying the predetermined FARin operation 630, the authentication apparatus determines whether acombination of the iris score and the face score, or a fusion scorecalculated by fusing the iris score and the face score, satisfies acombined authentication condition, in operation 650. In response to thecombined authentication condition being satisfied, the authenticationapparatus accepts the authentication.

In response to the combined authentication condition not beingsatisfied, the authentication apparatus acquires new iris data and facedata and performs the combined authentication process of FIG. 6A again.

In an example, in response to the condition being not satisfied inoperations 630 or 650, the authentication apparatus may determine thatthe authentication has failed by counting a predetermined number oftimes that the illustrated t of FIG. 6A is incremented, or the number oftimes either or both of the iris and face data are required, ordetermine that the authentication has failed upon a predetermined lengthof time since a beginning of the authentication operation havingexpired, for example. For example, in response to the condition notbeing satisfied in operation 630 or 650, the authentication apparatusaccumulates and stores the number of authentication failures resultingin the combined authentication process being performed again. Inresponse to the cumulative number of failures exceeding a predeterminedthreshold, the authentication apparatus may determine that theauthentication has failed. In another example, the authenticationapparatus accumulates and monitors the amount of time used or havingexpired for the combined authentication processes or entireauthentication process, and may determine that the authentication hasfailed in response to the cumulative monitored time exceeding apredetermined threshold. For example, in response to the condition notbeing satisfied in operation 630 or 650, the authentication apparatusdetermines whether the cumulative monitored time exceeds thepredetermined threshold. In an example, in response to a determinationthat the authentication has failed, the authentication apparatus mayprovide a user with a feedback indicating authentication failure, wheresuch feedback may be express or inherent, such as through expressfeedback that provides display, haptic, or audible feedback of failure(or success) or inherent feedback where the user does not gain access toan application or use of a device, or does not gain access to a securelocation, etc., noting that inherent success may similarly be indicatedthrough merely the user being given access to such an application or useof the device, etc.

FIG. 6B is a flowchart illustrating an example of a method withselective combined authentication. Referring to FIG. 6B, theauthentication apparatus extracts feature information of a modality, inoperation 610. The authentication apparatus sequentially performs asingle authentication and then selectively a combined authentication, inoperation 690. For example, the authentication apparatus may firstlyperform a face single authentication, selectively secondarily perform aniris single authentication, and selectively thirdly performs face andiris modalities combined authentication. In response to the either ofthe single authentications or the combined authentication beingsuccessful in each operation, the authentication apparatus mayimmediately determine that the authentication is successful, e.g.,without performing any remaining authentication(s).

In response to both the face single authentication and the iris singleauthentication having respectively failed, the authentication apparatusdetermines an entry condition related to whether to perform the combinedauthentication, prior to performing the combined authentication.

In response to the entry condition or another condition for the combinedauthentication not being satisfied, the authentication apparatus mayrepeat both the face and iris single authentication processes based onnew respective input data, e.g., through respective recapturings of suchrespective face and iris biometric information or through accessing ofstored face and/or iris biometric information subsequent to the alreadyinput data, such as where the respective cameras of the authenticationapparatus captures multiple frames. In a case in which authentication isstill not successful after the predetermined number of times that suchnew data is considered has been exceeded or after predetermined time haselapsed since the beginning of the authentication operation, theauthentication apparatus may finally determine that the authenticationhas failed.

FIG. 7 is a flowchart illustrating an example of a method with selectivecombined authentication. Referring to FIG. 7, a combined authenticationprocess that improves a user convenience and enhances a security at thesame time is illustrated.

As non-limiting examples, operations 610 through 650 of FIG. 6A includedetermining whether features or feature vectors corresponding to any onemodality match, e.g., within a predetermined similarity threshold,corresponding registered features or feature vectors with respect tothat one modality, and if a result of that determining indicates thatrecognition fails, there may be a further determination of whetherfeatures or feature vectors corresponding to any one combination ofmodalities, for example, a combination of a face modality and an irismodality, satisfy a condition for entering a combined authenticationand/or whether fused feature vectors satisfy a condition for anacceptance of the combined authentication, and based on that furtherdetermination the combined authentication may be selectively performed.Operations 710 through 750 of FIG. 7 include determining whether a“combi condition” and a “f(combi) condition” are satisfied with respectto various combinations of modalities. Hereinafter, for ease ofdescription, the condition for entering the combined authentication willbe denoted as “combi condition”, and the condition for the acceptance ofthe combined authentication of fused feature vectors will be denoted as“f(combi) condition”. Due to modalities being different the combicondition and/or the f(combi) condition are determined differently foreach combination of the corresponding different modalities.

In FIG. 7, depending on whether a security level of the combinedauthentication through each combination of the modalities is set to beexceedingly high, or the security level set for the combinedauthentication is set to not be high, one or more conditiondeterminations with respect to various authentication methods may beadded or omitted for each combination of the modalities in various suchexamples.

In response to data of many modalities being controlled to be input,e.g., as a result of controlled capturing of the corresponding biometricinformation, an authentication apparatus performs an authentication byvarious selective combinations of the modalities. For example, asdemonstrated in FIG. 7, the authentication apparatus may perform thecombined authentication up to M authentication methods includingrespective various combinations of at least one of N modalities. Thus, Mdenotes the number of authentication methods by a single authentication,two or more single authentications, the one or more singleauthentications and one or more combined authentications, or merely suchone or more combined authentications.

However, prior to performing the combined authentication by eachcombination of the modalities, the authentication apparatus determineswhether to perform the combined authentication based on whether scoress₁, s₂, . . . , s_(N) for the modalities satisfy the f(combi) conditionand/or the combi condition. The f(combi) condition and/or the combicondition are, for example, greater than or equal to, less than or equalto, less than, greater than a predetermined score, or a form of acombination thereof. In this example, the scores s₁, s₂, . . . , s_(N)for the modalities may correspond to feature distances or similarityscores between feature vectors corresponding to the modalities andenrollment feature vectors for the modalities stored in an enrollment DB705, as non-limiting examples. A condition with respect to the scoresfor the modalities may, thus, be used as an entry condition to determinewhether to enter the combined authentication.

The authentication apparatus may perform the combined authentication foreach combination of corresponding modalities in response to each of thescores for the modalities being input, e.g., rather than performing anauthentication after all the scores of the N modalities for the combinedauthentication are input completely. In an example, the authenticationapparatus may determine a criterion to determine whether to enter thecombined authentication based on a modality having a predeterminedexcellent security, i.e., a modality which has been predetermined toprovide a greatest or great security level compared to other modalities.

In operation 750, the authentication apparatus may determine whether afused score of a combination of first modalities satisfies a f₁(combi₁)condition based on the scores s₁, s₂, . . . , s_(N) for the modalities.In this example, the combination of the first modalities corresponds toa single modality, or a combination of at least two modalities. Further,the f₁(combi₁) condition corresponds to a single authenticationcondition in a case in which the combination of the first modalitiesincludes a single modality, and corresponds to a combined authenticationcondition in a case in which the combination of the first modalitiesincludes at least two modalities.

In response to the fused score of the combination of the firstmodalities not satisfying the f₁(combi₁) condition, the authenticationapparatus determines whether scores corresponding to a combination ofsecond modalities satisfy a combi₂ condition and/or whether fusedfeature vectors corresponding to the combination of the secondmodalities satisfy a f₂(combi₂) condition. As described above, examplesinclude such processes of determining a corresponding combi conditionand corresponding f(combi) condition with respect to variousauthentication methods, based on a predetermined security level of theauthentication, being added for each added combination of modalitiesbased on the predetermined security level. As another example, in a casein which the security level of the combined authentication ispredetermined to be low or a demand for the convenience is predeterminedhigh, examples include such processes of determining the correspondingcombi condition and corresponding f(combi) condition with respect to thevarious authentication methods being added or omitted for eachcombination of modalities based on such predetermined low security levelor high convenience.

FIG. 8 is a flowchart illustrating an example of a method with selectivecombined authentication. Referring to FIG. 8, in operation 810, anauthentication apparatus determines whether to perform a combinedauthentication based on any one or any combination of a first feature ofa first modality and a second feature of a second modality, for example.The authentication apparatus determines, for example, whether the firstfeature satisfies a first condition for a single authentication. Inresponse to the first feature not satisfying the first condition, theauthentication apparatus determines whether any one or any combinationof the first feature and the second feature satisfies a second conditiondifferent from the first condition. In response to the second conditionbeing satisfied, the authentication apparatus determines to perform thecombined authentication by a combination of the first modality and thesecond modality. The second condition is determined, for example, basedon a FAR of the first modality or the second modality.

In operation 820, the authentication apparatus generates a third featureby fusing the first feature and the second feature in response to adetermination to perform the combined authentication.

In operation 830, the authentication apparatus performs the combinedauthentication based on the third feature. In response to the thirdfeature satisfying a third condition for the combined authentication,the authentication apparatus performs the combined authentication.

FIG. 9 is a flowchart illustrating an example of a method with selectivecombined authentication. Referring to FIG. 9, in operation 910, anauthentication apparatus determines whether a first entry conditioncorresponding to a first combination of modalities is satisfied. Forexample, the authentication apparatus determines whether the first entrycondition is satisfied based on whether the first combination of themodalities satisfies a second condition different from a first conditionfor a single authentication. The first combination of the modalitiesincludes a first modality and a second modality different from the firstmodality. The authentication apparatus determines whether any one or anycombination of a first feature of the first modality and a secondfeature of the second modality satisfies the second condition. Inresponse to any one or any combination of the first feature and thesecond feature satisfying the second condition, the authenticationapparatus determines that the first entry condition is satisfied.

In operation 920, the authentication apparatus performs anauthentication by the first combination in response to the first entrycondition being satisfied.

In operation 940, the authentication apparatus determines whether asecond entry condition corresponding to a second combination of themodalities is satisfied in response to the authentication by the firstcombination being failed. In this example, the first entry condition andthe second entry condition are determined differently for eachcombination of the modalities.

In operation 940, the authentication apparatus performs anauthentication by the second combination in response to the second entrycondition being satisfied.

FIG. 10 is a flowchart illustrating an example of a method withselective combined authentication. Referring to FIG. 10, a combinedauthentication process is illustrated. Operations 1010 through 1050 ofFIG. 10 may be considered similar to operations 710 through 750 of FIG.7, for example, with some differences from those of FIG. 7.

For example, unlike FIG. 7, in response to data of many modalities beinginput, an authentication apparatus determines whether a combination offirst modalities satisfies a combi₁ condition based on scores s₁, s₂, .. . , s_(N) for the modalities, in operation 1030. In response to thecombination of the first modalities not satisfying the combi₁ condition,the authentication apparatus determines whether a combination of secondmodalities satisfies a combi₂ condition. In response to each of thecombinations of the modalities not satisfying the combi condition, theauthentication apparatus continues a comparison of a combination of newmodalities with the combi condition until the combi condition issatisfied.

For example, in a case of combi_(I) condition=(M₁, M₂, M₃) in operation1030, an entry condition corresponding to the combi_(I) combination isdetermined, for example, by an M₁ condition, an M₂ condition, an M₃condition, or various combinations thereof.

In response to the combination of the first modalities satisfying thecombi₁ condition, the authentication apparatus determines whether afused score of the combination of the first modalities satisfies af(combi₁) condition, in operation 1050. In this example, a score foreach modality is used to determine whether to enter a combinedauthentication. In response to the fused score of the combination of thefirst modalities not satisfying the f(combi₁) condition, theauthentication apparatus determines whether the combination of thesecond modalities satisfies a combi₂ condition. In response to the fusedscore of the combination of the first modalities satisfying thef(combi₁) condition, the authentication apparatus accepts the combinedauthentication.

FIG. 11 is a flowchart illustrating an example of a method withselective combined authentication. Referring to FIG. 11, a process ofcontrolling an IR camera and a color camera to capture an IR image and acolor face image, e.g., controlling an input of the IR image and thecolor face image to the authentication apparatus, and performing acombined authentication using the IR image and the color face image isillustrated. For ease of description, an application using a color imagewill be described. However, examples are not limited to such a scenario,and are also substantially similarly or identically applicable to anapplication using a monochrome image or a depth image or 3D image, asnon-limiting examples.

The authentication apparatus matches a plurality of features or featurevectors of an iris modality obtained from the IR image and a pluralityof features or feature vectors corresponding to a face modality obtainedfrom the color face image through a feature extraction process ofoperation 1110 with enrollment feature vectors for modalities enrolledin an enrollment DB. The authentication apparatus calculates scores i₁,i₂, . . . , i_(N) corresponding to the plurality of features of the irismodality and scores f₁, f₂, . . . , f_(O) corresponding to the pluralityof features of the face modalities through operation 1110. In thisexample, the plurality of features of the iris modality include aHamming distance of an iris, a bit count of the iris, a radius of theiris, a shape and a color of the iris, and morphemes of retinalcapillaries. Further, the plurality of features corresponding to theface modality include a partial face and an entire face, and may includefeatures from each of the partial face and the entire face.

In operation 1130, the authentication apparatus determines whether acombination of first modalities satisfies a combi₁ condition based onthe scores i₁, i₂, . . . , i_(N), and f₁, f₂, . . . , f_(O) for themodalities. The combination of the first modalities is, for example, acombination of the Hamming distance of the iris and the partial face. Inresponse to the combination of the first modalities not satisfying thecombi₁ condition, the authentication apparatus determines whether a newcombination of second modalities satisfies a combi₂ condition. Thecombination of the second modalities is, for example, a combination ofthe morphemes of retinal capillaries and the entire face.

In response to the combination of the second modalities satisfying thecombi₂ condition, the authentication apparatus determines a fused scoreof the combination of the second modalities satisfies a f₂(combi₂)condition. In response to the fused score of the combination of thesecond modalities satisfying the f₂(combi₂) condition, theauthentication apparatus accepts a combined authentication by thecombination of the second modalities.

In an example, the authentication apparatus may selectively control thecombi condition to be always true or to be always false for somesituations. For example, while the authentication apparatus may beconfigured to perform the determination of the combi condition, theauthentication apparatus may selectively control the combi₁ conditioncorresponding to a face score f₁ among scores corresponding to the facemodality to be always true, thereby always performing a singleauthentication of a face score, irrespective of whether the combi₁condition would otherwise have been satisfied with respect to the facescore f₁. In another example, the authentication apparatus mayselectively control the combi₁ condition corresponding to the face scoref₁ to be always false, thereby skipping the single authentication of theface score, irrespective of whether the combi₁ condition would otherwisehave been satisfied.

Further, in addition to the above examples, the authentication apparatusmay perform an authentication considering both user convenience andsecurity by combining the combi condition and the f(combi) condition invarious manners. For example, f(combi) conditions may be set to performsingle authentication(s) based on individual features and combinedauthentication(s) may be based on a combination of a plurality offeatures in a predetermined order. A combi condition corresponding toeach f(combi) condition is set based on a condition different from asingle authentication condition used for the f(combi) condition. Also,as noted above, in some cases, the combi condition may be controlled tobe true or false irrespective of whether the corresponding combicondition would otherwise have resulted in the true or falsedetermination.

FIG. 12 illustrates an example of a user interface, e.g., a mobiledevice including the user interface. Referring to FIG. 12, when a useruses a biometric authentication, a type of a modality that is being, oris going to be, used to unlock a smart phone is displayed on a lockscreen of the smart phone. For example, in a case in which the modalitythat is being, or is going to be, used to unlock the smart phone is aface modality, a face icon is displayed on the lock screen, as shown ina first screen 1210. In a case in which the modality that is being, oris going to be, used to unlock the smart phone is an iris modality, aniris icon is displayed on the lock screen, as shown in a second screen1230. In a case in which the modality that is being, or is going to be,used to unlock the smart phone is a combination of the face modality andthe iris modality, the face icon of the first screen 1210 and the irisicon of the second screen 1230 are both displayed or alternatelydisplayed on the lock screen.

In an example, a first preview screen for inputting a face image and asecond preview screen for inputting an iris image may additionally beselectively displayed, though examples exist where such preview screensare not displayed. The preview screens for inputting the face image orthe iris image may be feedback images of the user guiding the user toposition their face or their eyes in certain positions relative to thecameras, as non-limiting examples. Whether to display the first previewscreen and whether to display the second preview screen may be set invarious manners. For example, in a case of using a single authenticationof the face modality, the first preview screen may be controlled to notbe displayed. However, in a case of using a single authentication of theiris modality, the second preview screen may be controlled to bedisplayed. Further, in a case of using a combined authentication of theface modality and the iris modality, both the first preview screen andthe second preview screen may be controlled to not be displayed, bothmay be controlled to be displayed, or only the second preview screen maybe controlled to be displayed.

Although FIG. 12 illustrates the face modality and the iris modality,examples exist where other modalities, such as a fingerprint modality,are additionally or alternatively implemented, i.e., alternatively toeither or both of the face image and the iris image modalities, such asfor the example unlocking operation, noting that the example of FIG. 12is not limited to only such an unlocking operation.

FIGS. 13A and 13B illustrate an example of a biometrics registrationprocess. Referring to FIG. 13A, in operation 1310, a user registers aface image using an image sensor of or attached to a smart phone. Inoperation 1330, the user registers an iris image using an IR sensor ofor attached to the smart phone. If the user is determined to be wearingglasses or merely as a reminder to the user, the operation 1330 mayinclude requesting the user take off the glasses to register the irisimage.

In an example, the user may select the type(s) of a modality to be usedfor an authentication for various predetermined applications orfunctions. Thus, for example, the user may change the type of the usermodality, e.g., from a default or previous setting, to be used to unlockthe smart phone in settings of the smart phone.

Referring to FIG. 13B, a screen for selecting a type of a modality to beused for a biometric authentication is illustrated. For example, optionsto be selected include a first option 1370 corresponding to a singleauthentication using a face modality, a second option 1390 correspondingto a single authentication using an iris modality, and a third option1350 corresponding to a combined authentication using a combination ofthe face modality and the iris modality. In response to the first option1370 being selected, the face modality is used for the biometricauthentication. In response to the second option 1390 being selected,the iris modality is used for the biometric authentication. In responseto the third option 1350 being selected, both the face modality and theiris modality are used for the biometric authentication. The firstoption 1370, the second option 1390, and the third option 1350 areselected exclusively from each other. For example, in response to anyone option being selected, an option selected before is automaticallycancelled. As another example, such different options may berepresentative of offering the user to select between options of varyinguser convenience and security.

In an example, the third option 1350 corresponding to the combinedauthentication may not be displayed as a separate selectable option, butrather, the user may select this third option by the first option 1370and the second option 1390 being selected concurrently. In a case inwhich the first option 1370 and the second option 1390 are selectedconcurrently, the combined authentication using the combination of theface modality and the iris modality is performed for the biometricauthentication.

Although a case of using the face modality and the iris modality for thebiometric authentication is described in FIG. 13B, examples are notlimited thereto. Various examples exist with additional or alternativemodalities. For example, the user may be provided with selectableoptions for at least three modalities, such as in various selectablecombinations of face, iris, fingerprint, vein, etc., modalities.

In response to the type of the modality being set, the authenticationapparatus verifies whether enrollment data corresponding to the type ofthe modality is stored. In response to verification that the enrollmentdata corresponding to the type of the modality is stored, theauthentication apparatus changes the type of the modality for theauthentication immediately without performing an additional registrationprocess. If the enrollment data corresponding to the type of themodality is not stored, the authentication apparatus requests the userprepare for the capturing of the corresponding enrollment data, and thatenrollment data corresponding to the type of the modality is capturedand stored as registration data.

In an example, the combined authentication of the face modality and theiris modality may be set by the user in a situation in which pre-storedenrollment data for the same is absent. In this example, theauthentication apparatus captures or acquires a face image and an irisimage, extracts respective features or feature vectors from the capturedor acquired face image and iris image, and respectively stores the sameas enrollment data of the face modality and enrollment data of the irismodality.

In another example, the combined authentication of the face modality andthe iris modality may be set by the user in a situation in which onlyenrollment data of the face modality is already stored. In this example,the authentication apparatus captures or acquires an iris image only,extracts features or feature vectors from the captured or acquired irisimage, and stores the extracted features or feature vectors asenrollment data of the iris modality.

In a case in which the user changes such settings from the combinedauthentication of the face modality and the iris modality to the singleauthentication of the iris modality, the authentication apparatus maydetermine whether enrollment data of the iris modality is stored. Inthis example, since the enrollment data of the iris modality is alreadystored, the authentication apparatus changes the settings to use thesingle authentication of the iris modality without performing anadditional registration process.

In an example, enrollment data may be managed for each modality. Forexample, the user may selectively delete enrollment data of apredetermined modality among the pre-stored enrollment data, such aswhere selectable options for each of the modalities is displayed and theuser may select one or more of such options to control the deletions ofthe corresponding enrollment data. In this example, after such adeletion of a select one or more enrollment data for corresponding oneor more modalities, the authentication apparatus changes anauthentication type based on a currently set authentication type and theremaining enrollment data. For example, enrollment data of the facemodality and enrollment data of the iris modality may be already stored,and the currently set authentication type may be a combinedauthentication. If the enrollment data of the iris modality is deleted,the authentication apparatus may automatically change the authenticationtype to the single authentication of the face modality. In anotherexample, when all the enrollment data of biometric modalities aredeleted, the authentication type may be automatically changed to a typeof performing an authentication through another process, such as apattern or a password entry based on user set pattern and password.

While the above examples are discussed with respect to mobile devices,e.g., a smart phone or tablet, these examples are also representative ofother devices other than such a smart phone or tablet, and alsoapplicable to modalities other than the face modality and/or the irismodality.

FIG. 14 illustrates an example of performing a selective combinedauthentication by utilizing a correlation between different modalities.In this example, the authentication apparatus may utilize a determinedcorrelation between a first image and a second image, for example, in aprocess of acquiring a first image for a first modality and a secondimage for a second modality.

For example, in an example where the first modality is a face modalityand the second modality is an iris modality, an eye region in the firstimage may be expected by the authentication apparatus to have arelatively high correlation with the second image. In this example, inresponse to a single authentication using the first image having failed,the authentication apparatus may determine whether the eye region isdetected in the first image, and use the result of that eye regiondetection determination to determine whether to perform a singleauthentication using the second image or a combined authentication usingthe first image and the second image. In detail, in the authenticationapparatus, a configured angle of view of an image sensor for the firstimage may be greater than a configured angle of view of an IR sensor forthe second image. In this example, if an eye region is not detected fromthe first image, there may be a high probability of the second image notincluding iris information. When iris information is not included in thesecond image, it may be impossible to perform the single authenticationusing the second image or the combined authentication. Thus, theauthentication apparatus may control a recapturing or reacquiring of thefirst image, instead of acquiring the second image, thereby retrying thesingle authentication using the newly captured or acquired first image.

Although the above example was explained using a discussion of the firstimage being a color image 1410 and the second image being an IR image1420, as illustrated in FIG. 14, examples are not limited thereto. In anexample, the first image may be the IR image and the second image may bethe color image, or both the first image and the second image may be IRimages, or both the first image and the second image may be colorimages. In another example, the first image may be a depth image and thesecond image may be a color image. Thus, as non-limiting examples, thefirst image may be one of a color image, an IR image, and a depth image,and the second image may also be one of a color image, an IR image, anda depth image.

The authentication apparatus captures or acquires the color image 1410as the first image. For example, the first image sensor of theauthentication apparatus generates the color image 1410 by capturing aface of a person as an object, as shown in FIG. 14.

In response to a single authentication using the color image 1410 havingfailed, the authentication apparatus identifies a landmark point of theobject with respect to the color image 1410. For example, theauthentication apparatus extracts a facial feature point of the personfrom the color image 1410 based on an object model. In an example, theauthentication apparatus checks whether a landmark point is identifiedwithin a predetermined region 1411. Example landmark points arerepresented as respective dots (1431, 1432, 1433, 1434, 1435, and 1436)in FIG. 14. In this example, landmark points 1431, 1433, 1434, and 1436may correspond to the respective lateral extents of the eyes, andlandmarks 1432 and 1435 may correspond to the respective pupils of theeyes, as non-limiting examples. The predetermined region 1411 of thecolor image 1410 may be a region having extents defined by an angle ofview of the IR image 1420, and thus the region 1411 may correspond tothe angle of the view of the IR image 1420. In a case in which alandmark is not detected in the predetermined region 1411 of the colorimage 1410, the authentication apparatus predicts that iris informationis not included in the IR image 1420. The landmark may include any ofvarious facial landmarks, such as the aforementioned example elements ofthe eye(s), as well as respective elements of eye brows, nose, mouth,jawline, etc., as non-limiting examples.

FIG. 15 is a block diagram illustrating an example of an authenticationapparatus with selective combined authentication. Referring to FIG. 15,an authentication apparatus 1500 includes the processor 1510. Theauthentication apparatus 1500 further includes a memory 1530, acommunication interface 1550, and sensors 1570. The processor 1510, thememory 1530, the communication interface 1550, and the sensors 1570communicate with each other through a communication bus 1505.

The processor 1510 performs a single authentication based on a firstmodality among multiple modalities. In response to the singleauthentication having failed, the processor 1510 determines whether toperform a combined authentication by a combination of some or all of themodalities based on a second condition different from a first conditionfor the single authentication. The processor 1510 performs the combinedauthentication in response to a determination to perform the combinedauthentication. The processor may thus be configured to perform any one,any combination, or all operations described above with respect to FIGS.1-14.

The memory 1530 includes an enrollment DB containing registrationfeature vectors, for example, for the respective modalities, and mayfurther store fusion registration feature vectors as registered fusedfeature vectors corresponding to the aforementioned fusion examples. Theenrollment DB may correspond to, for example, the enrollment DB 410 ofFIG. 4. The memory 1530 may be a volatile memory or a non-volatilememory.

The communication interface 1550 may indicate a single authenticationresult and/or a combined authentication result, such as by controllingan express display of the success (and/or failure) of the authenticationon a display device 1580 of the authentication apparatus 1500 or aremote display outside of the authentication apparatus 1500. As notedabove, the indication may also be inherent, such as through theauthentication apparatus 1500 unlocking the display, or providing accessto additional functions of an application, for indications of success,or the prevention of such unlocking or success for authenticationfailures The communication interface 1550 may also transmit the successor failure to another device, such as for secure entry to a secure area.The communication interface 1550 may also receive one or more modalitiesfrom outside of the authentication apparatus 1500, and/or receiveinformation related to an environment in which a modality is collectedfrom a user.

The sensors 1570 include, for example, an image sensor, an IR sensor, afingerprint recognition sensor, and a voice recognition sensor. Thesensors collect various modalities. In addition, the sensors may includean environmental sensor that may capture and provide information on theenvironment in which one or more modalities are collected, such as anambient light level sensor.

In an example, the processor 1510 determines whether to perform thecombined authentication based on any one or any combination of a firstfeature of the first modality and a second feature of a second modality.The processor 1510 generates a third feature by fusing the first featureand the second feature in response to the determination to perform thecombined authentication, and performs the combined authentication basedon the third feature.

In another example, the processor 1510 determines whether a first entrycondition corresponding to a first combination of the modalities issatisfied, and performs an authentication by the first combination inresponse to the first entry condition being satisfied. In response tothe authentication by the first combination being failed, the processor1510 determines whether a second entry condition corresponding to asecond combination of the modalities is satisfied, and performs anauthentication by the second combination in response to the second entrycondition being satisfied.

In addition, the processor 1510 may be configured to perform one or moreor all operations described with reference to FIGS. 1 through 14 byexecution of instructions, e.g., a program or computer readable code,stored in the memory 1530. In an example, the authentication apparatus1500 is connected to an external device, for example, a personalcomputer or a network, through the communication interface 1550, andthereby exchanges data with the external device, which may include theaforementioned example one or more modalities. The authenticationapparatus 1500 may be a smart television, a smart phone, a smartvehicle, and various electronic systems, as non-limiting examples.

The authentication apparatuses, the processors, the enrollment DBmemory, the matcher 420, authenticator 440, the entry conditiondeterminer 430, the processor 450, the authentication apparatus 1500,the processor 1510, the memory 1530, the bus 1505, the communicationinterface 1550, the sensors 1570, and the display 1580 and any otherapparatuses, units, modules, devices, and other components describedherein are implemented by hardware components. Examples of hardwarecomponents that may be used to perform the operations described in thisapplication where appropriate include controllers, sensors, generators,drivers, memories, comparators, arithmetic logic units, adders,subtractors, multipliers, dividers, integrators, and any otherelectronic components configured to perform the operations described inthis application. In other examples, one or more of the hardwarecomponents that perform the operations described in this application areimplemented by computing hardware, for example, by one or moreprocessors or computers. A processor or computer may be implemented byone or more processing elements, such as an array of logic gates, acontroller and an arithmetic logic unit, a digital signal processor, amicrocomputer, a programmable logic controller, a field-programmablegate array, a programmable logic array, a microprocessor, or any otherdevice or combination of devices that is configured to respond to andexecute instructions in a defined manner to achieve a desired result. Inone example, a processor or computer includes, or is connected to, oneor more memories storing instructions or software that are executed bythe processor or computer. Hardware components implemented by aprocessor or computer may execute instructions or software, such as anoperating system (OS) and one or more software applications that run onthe OS, to perform the operations described in this application. Thehardware components may also access, manipulate, process, create, andstore data in response to execution of the instructions or software. Forsimplicity, the singular term “processor” or “computer” may be used inthe description of the examples described in this application, but inother examples multiple processors or computers may be used, or aprocessor or computer may include multiple processing elements, ormultiple types of processing elements, or both. For example, a singlehardware component or two or more hardware components may be implementedby a single processor, or two or more processors, or a processor and acontroller. One or more hardware components may be implemented by one ormore processors, or a processor and a controller, and one or more otherhardware components may be implemented by one or more other processors,or another processor and another controller. One or more processors, ora processor and a controller, may implement a single hardware component,or two or more hardware components. A hardware component may have anyone or more of different processing configurations, examples of whichinclude a single processor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods described with respect to FIGS. 1-15 that perform theoperations described in this application are performed by computinghardware, for example, by one or more processors or computers,implemented as described above executing instructions or software toperform the operations described in this application that are performedby the methods. For example, a single operation or two or moreoperations may be performed by a single processor, or two or moreprocessors, or a processor and a controller. One or more operations maybe performed by one or more processors, or a processor and a controller,and one or more other operations may be performed by one or more otherprocessors, or another processor and another controller. One or moreprocessors, or a processor and a controller, may perform a singleoperation, or two or more operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computer using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions in the specification, which disclosealgorithms for performing the operations that are performed by thehardware components and the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access programmable readonly memory (PROM), electrically erasable programmable read-only memory(EEPROM), random-access memory (RAM), dynamic random access memory(DRAM), static random access memory (SRAM), flash memory, non-volatilememory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs,DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-rayor optical disk storage, hard disk drive (HDD), solid state drive (SSD),flash memory, a card type memory such as multimedia card micro or a card(for example, secure digital (SD) or extreme digital (XD)), magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and provide the instructions or software and any associated data,data files, and data structures to one or more processors or computersso that the one or more processors or computers can execute theinstructions. In one example, the instructions or software and anyassociated data, data files, and data structures are distributed overnetwork-coupled computer systems so that the instructions and softwareand any associated data, data files, and data structures are stored,accessed, and executed in a distributed fashion by the one or moreprocessors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents. Therefore, the scope of the disclosure is defined not bythe detailed description, but by the claims and their equivalents, andall variations within the scope of the claims and their equivalents areto be construed as being included in the disclosure.

What is claimed is:
 1. A processor-implemented authentication method,the method comprising: performing a single authentication based on afirst modality among plural modalities; and in response to the singleauthentication having failed, determining whether to perform a combinedauthentication by a combination of two or more of the plural modalities,and selectively, depending on a result of the determining of whether toperform the combined authentication, performing the combinedauthentication.
 2. The method of claim 1, wherein the determining ofwhether to perform the combined authentication comprises determiningwhether a second condition for the combined authentication is satisfied,the second condition being different from a first conditiondeterminative of success or failure of the single authentication.
 3. Themethod of claim 2, wherein the second condition is determineddifferently for each different combination of the plural modalities. 4.The method of claim 2, wherein the second condition is determined basedon a false acceptance rate (FAR) of a modality representing a highestsecurity among the plural modalities.
 5. The method of claim 2, whereinthe second condition is determined based on a FAR of a modalityrepresenting a highest convenience among the plural modalities.
 6. Themethod of claim 1, wherein the performing of the single authenticationcomprises determining whether a first feature of the first modalitysatisfies a first condition for the single authentication.
 7. The methodof claim 1, wherein the determining of whether to perform the combinedauthentication comprises determining whether a first feature of thefirst modality satisfies a second condition different from a firstcondition for the single authentication.
 8. The method of claim 1,wherein the two or more of the plural modalities include the firstmodality and a second modality that is different from the firstmodality, and the determining of whether to perform the combinedauthentication comprises determining whether a first feature of thefirst modality, a second feature of the second modality, or acombination of the first feature and the second feature satisfies asecond condition different from a first condition for the singleauthentication.
 9. The method of claim 1, wherein the two or more of theplural modalities include the first modality and a second modality, andwherein the performing of the combined authentication comprises:generating a third feature by fusing a first feature of the firstmodality and a second feature of the second modality, in response to adetermination to perform the combined authentication as a result of thedetermining of whether to perform the combined authentication; andperforming the combined authentication based on the third feature. 10.The method of claim 9, wherein the performing of the combinedauthentication comprises determining whether the third feature satisfiesa third condition for the combined authentication.
 11. The method ofclaim 1, wherein the combination of the two or more of the pluralmodalities is determined based on a determined security or a conveniencelevel for the combined authentication.
 12. The method of claim 1,wherein the plural modalities include any one or any combination of aface image modality, a fingerprint image modality, an iris imagemodality, a vein image modality, a palmprint image modality, a signaturemodality, a voice modality, a gait modality, and a DNA structuremodality of a user.
 13. A processor-implemented authentication method,the method comprising: determining whether to perform a combinedauthentication, considering a first modality and a second modality,based on any one or any combination of a first feature of the firstmodality and a second feature of the second modality; and selectively,based on a result of the determining, performing the combinedauthentication based on the first feature and the second feature. 14.The method of claim 13, wherein the performing of the combinedauthentication comprises: generating a third feature by fusing the firstfeature and the second feature; and performing the combinedauthentication based on the third feature.
 15. The method of claim 14,wherein the performing of the combined authentication based on the thirdfeature comprises determining whether the third feature satisfies athird condition for the combined authentication.
 16. The method of claim13, wherein the determining comprises: determining whether the firstfeature, the second feature, or a combination of the first feature andthe second feature satisfies a second condition different from a firstcondition for a performed single authentication that is based on thefirst modality or the second modality; and determining to perform thecombined authentication in response to the second condition beingsatisfied.
 17. The method of claim 16, wherein the second condition isdetermined based on a first false acceptance rate (FAR) of the firstmodality, a second FAR of the second modality, or a combination of thefirst FAR and the second FAR.
 18. A processor implemented authenticationmethod, the method comprising: determining whether a first entrycondition corresponding to a first combination of plural modalities issatisfied; selectively, depending on a result of the determining ofwhether the first entry condition is satisfied, performing anauthentication by the first combination; and in response to theauthentication by the first combination having failed, determiningwhether a second entry condition corresponding to a second combinationof the plural modalities is satisfied, and selectively, depending on aresult of the determining of whether the second entry condition issatisfied, performing an authentication by the second combination. 19.The method of claim 18, wherein the first entry condition and the secondentry condition are respectively determined differently for each of thefirst combination of the plural modalities and the second combination ofthe plural modalities.
 20. The method of claim 18, wherein thedetermining of whether the first entry condition is satisfied comprisesdetermining whether the first entry condition is satisfied based onwhether the first combination of the plural modalities satisfies asecond condition that is different from a first condition for aperformed single authentication of a modality of the plural modalities.21. The method of claim 20, wherein the first combination of the pluralmodalities includes a first modality and a second modality that isdifferent from the first modality, and the determining of whether thefirst entry condition is satisfied comprises: determining whether anyone or any combination of a first feature of the first modality and asecond feature of the second modality satisfies the second condition;and determining that the first entry condition is satisfied in responseto the second condition being satisfied.
 22. A processor implementedauthentication method, the method comprising: performing a singleauthentication based on a first modality among plural modalities; andselectively, dependent on the single authentication having failed,performing a combined authentication by a combination of the pluralmodalities.
 23. The method of claim 22, further comprising: in responseto the single authentication having failed, performing another singleauthentication based on a second modality among the plural modalities,and wherein the selective performing of the combined authenticationincludes determining whether to perform the combined authentication inresponse to the other single authentication having failed.
 24. Themethod of claim 22, wherein the performing of the combinedauthentication comprises: generating a third feature by fusing a firstfeature of the first modality and a second feature of a second modalityamong the plural modalities; and performing the combined authenticationbased on the third feature.
 25. A biometric authentication method ofauthenticating a user using a first biometric modality and a secondbiometric modality which are different from each other, the biometricauthentication method comprising: determining whether biometricinformation of the user satisfies one of a first condition correspondingto a feature of the first biometric modality and a second conditioncorresponding to a feature of the second biometric modality; determiningwhether the biometric information of the user satisfies a combinedcondition, for a combined authentication, corresponding to the featureof the first biometric modality and the feature of the second biometricmodality; and determining that the combined authentication is successfulin response to a determination that the biometric information of theuser satisfies one of the first condition and the second condition, andsatisfies the combined condition.
 26. The biometric authenticationmethod of claim 25, wherein the combined condition is a conditioncorresponding to a feature resulting from a fusing of the feature of thefirst biometric modality and the feature of the second biometricmodality.
 27. The biometric authentication method of claim 25, whereinthe combined condition is a condition combining a score calculated basedon the feature of the first biometric modality and a score calculatedbased on the feature of the second biometric modality.
 28. The biometricauthentication method of claim 25, wherein the determining of whetherthe biometric information of the user satisfies the combined conditioncomprises: generating a third feature by fusing the feature of the firstbiometric modality and the feature of the second biometric modality,where the combined condition corresponds to the third feature; anddetermining whether the combined condition corresponding to the thirdfeature is satisfied.
 29. The biometric authentication method of claim25, wherein the determining of whether the biometric information of theuser satisfies the combined condition comprises: generating a thirdscore by combining a first score calculated based on the feature of thefirst biometric modality and a second score calculated based on thefeature of the second biometric modality; and determining whether thecombined condition is satisfied based on the third score.
 30. Thebiometric authentication method of claim 25, wherein the first biometricmodality is an iris modality, and the second biometric modality is aface modality.